Using Augmented Reality to Tutor Military Tasks in the Wild

Intelligent Tutoring Systems (ITSs) have been shown to be effective in training a variety of military tasks. However, these systems are often limited to laboratory settings on standard PCs and laptops which focus on exercising cognitive skills (e.g., decision-making and problem solving) and may potentially limit the learning and retention of the dismounted Soldiers and Marines training to master physical tasks. Augmented reality presents the possibility of combining intelligent tutoring with hands-on applications in realistic physical environments. In this paper, we examine the use of an augment-reality based adaptive tutoring system for instruction in the wild, locations where no formal training infrastructure is present, and identify the challenges that arise when developing such a system. We began the transition from desktop tutoring to the wild by exploring an existing real life mockup of a market scene along with low cost commercial-off-the-shelf devices (e.g., HMDs coupled with depth cameras) and a 3D model of the environment. The goal of our canning approach is to use “human in the loop” 3D scene acquisition via augmented reality so that the scene can be scanned efficiently and with complete coverage. Using this 3D model, intelligent tutoring systems can adaptively manage instruction while being aware of the physical and augmented objects in the scenario. Furthermore, with this awareness of the physical environment, we hope to provide augmented effects and objects (e.g., virtual humans) that register to the physical environment and respond realistically to interactions with the trainee. We also explored developing a training scenario for evaluation of our system that is made to work with emerging low-cost commercial augmented reality devices (e.g., Epson Moverio). Our approach examines the merging of intelligent tutoring with augmented reality to be used for hands-on immersive training of psychomotor tasks in a setting beyond the typical desktop tutoring session.